1. Field of the Invention
The invention relates to a resolution enhancement apparatus and method, which convert image data captured by a camera or received by a television into image data with a higher resolution.
2. Description of the Related Art
Televisions and displays with large numbers of pixels, i.e., with high resolutions, are highly popular. Upon displaying an image, a television or display converts the number of pixels of image data into that of a panel. Especially, in resolution enhancement conversion that increases the number of pixels, a multi-frame degradation inverse conversion method is known as a method of obtaining an image sharper than a linear interpolation method (for example, see JP-A 2000-188680 (KOKAI) (pp. 3-7, FIG. 22), and S. Park, et. al. “Super-Resolution Image Reconstruction: A Technical Overview,” IEEE Signal Processing Magazine, USA, IEEE, May 2003, p. 21-36).
The multi-frame degradation inverse conversion method pays attention Lo the fact that an object which appears in a reference frame also appears in another frame, and detects a motion of the object with high precision of a pixel interval or less, thus enhancing the resolution by calculating a plurality of sampling values whose positions deviate very little from an identical local portion of the object.
The multi-frame degradation inverse conversion method will be described in more detail below. This method sequentially converts a time-series sequence of low-resolution frames into high-resolution frames. For example, three time-series frames of a moving image obtained by capturing a moving automobile are used as low-resolution images, and resolution enhancement is attained using one frame from these three frames as a reference frame. For example, the resolution of this frame is enhanced to ×2 in the vertical direction and ×2 in the horizontal direction. With respect to unknown pixels of a high-resolution image, pixels of a low-resolution image, i.e., known sampling values are sparse. In this state, the pixel values of the high-resolution image can be estimated. However, when the number of known sampling values is increased in advance, a more precise high-resolution image can be obtained. For this purpose, the multi-frame degradation inverse conversion method detects a position, in an image plane of the reference frame, of an object which appears at a given pixel position of a low-resolution image other than the reference frame, and uses that pixel value as a sampling value at the corresponding point in the reference frame.
More specifically, for example, a square block, which includes a given pixel as the center, and one side of which is defined by several pixels, is extracted from a low-resolution image, and the reference frame is searched for a portion which has a size the same as this block and includes pixel values close to those of the extracted block. This search is conducted at a sub-pixel precision (for example, see Shimizu and Okutomi, “Significance and Attributes of Sub-Pixel Estimation on Area-Based Matching”, The transactions of the Institute of Electronics, Information and Communication Engineers, D-II, the Institute of Electronics, Information and Communication Engineers, December 2002, Vol. 85, No. 12, pp. 1791-1800). The center of the found corresponding block is defined as a corresponding point. In this manner, a point A of an image plane corresponding to another frame is associated with a point B of an image plane corresponding to the reference frame as an identical position of an identical object. This association is expressed by a motion vector having the point A as a start point and the point B as an end point. Since the search is done at a sub-pixel precision, generally the start point of the motion vector is on a pixel position, and the end point is on a position where no pixel exists. Such motion vector is calculated for all pixels of the low-resolution image, and motion vectors to the reference frame having each pixel as the start point are similarly detected for other low-resolution images. Next, the pixel values of the start points are allocated as sampling values at the end points of the respective motion vectors. Finally, uniformly allocated pixel values of a high-resolution image are calculated from the nonuniformly allocated sampling values. As this scheme, nonuniform interpolation, a POCS method, and the like are known (for example, see JP-A 2000-186680 (KOKAI) (pp. 3-7, FIG. 22), and S. Park, et. al. “Super-Resolution Image Reconstruction: A technical Overview,” IEEE Signal Processing Magazine, USA, IEEE, May 2003, p. 21-36).
As described above, the multi-frame degradation inverse conversion method can produce a sharp high-resolution image when an image captured by a camera is input without any modification. However, when an image like a received television image, which is sampled by a camera and then undergoes image compression, noise removal filtering, and the like is input, since pixel values have changed from sampling values, the method cannot produce a sufficiently sharp image.